Methods and systems for augmented reality safe visualization during performance of tasks

ABSTRACT

Systems and methods for augmented reality (AR) safe visualization for use with a near to eye (NTE) display system worn by a user are provided. The system includes: a processor programmed with an AR program and a task database storing task data; and, a camera mounted to the NTE display system and providing video input. The processor receives the video input and coordinates video image processing of the video input to identify therein a user&#39;s hand and an object. The processor receives an intended task from the user and retrieves associated task data based thereon. The processor processes the task data with the intended task to render a visualized item, such as a job card. The processor determines when the visualized item is in front of the hand while the user is performing the task and removes the visualized item responsive thereto.

TECHNICAL FIELD

The technical field generally relates to computer vision systems, andmore particularly relates to systems and related operating methods foraugmented reality safe visualization during performance of tasks.

BACKGROUND

Augmented reality (AR) technology, as may be used with near to eyedisplays (NTE) and smart glasses, create new benefits and opportunitiesfor visualization of objects and task instructions associated with themaintenance of a device or equipment. Visualized objects and taskinstructions may be rendered on a see-through display using either two-or three-dimensional conventional AR systems. Therefore, two- andthree-dimensional AR can enable a user to navigate through one or moretasks quite easily.

However, the use of conventional AR systems during performance of a taskpresents some limitations and technological problems. Among the mostcritical is the safety of the user during performance of the task. Forexample, during a maintenance task, it is desirable that visualizationbe done in a manner that does not interfere with a user's view of hishands or tools and does not otherwise disturb the user's vision. Theprovided systems and methods for augmented reality safe visualizationduring performance of tasks improve upon existing AR systems byaddressing these technological problems in an unconventional way, inaddition to providing other technological enhancements.

BRIEF SUMMARY

This summary is provided to describe select concepts in a simplifiedform that are further described in the Detailed Description. Thissummary is not intended to identify key or essential features of theclaimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

Provided is an augmented reality (AR) safe visualization method for usewith a near to eye (NTE) display system worn by a user. The methodincludes: at a processor associated with the NTE display system andprogrammed with an AR program, receiving video input; receiving anintended task; retrieving, for the intended task, an object, a tool, anda job card, from a task database; rendering a visualized job card,visualized object, and a visualized tool on the NTE display system;video image processing the video input to identify and locate a user'shand and the object; determining that the user is performing theintended task; determining when there is a job card block, defined aswhen the visualized job card is in front of the hand; determining whenthere is a tool block, defined as when the visualized tool is in frontof the hand; determining when there is an object block, defined as whenthe visualized object is in front of the hand; removing the visualizedjob card, or a portion of the visualized job card in front of the hand,when there is a job card block; removing the visualized tool, or aportion of the visualized tool in front of the hand, when there is atool block; and removing the visualized object, or a portion of thevisualized object in front of the hand, when there is an object handblock.

Another method for augmented reality (AR) safe visualization for usewith a near to eye (NTE) display system worn by a user is provided. Themethod includes: at a processor associated with the NTE display systemand programmed with an AR program, performing the operations of:receiving video input from a camera mounted to the NTE display system;receiving an intended task from a user input device; retrieving taskdata from a task database based on the intended task; video imageprocessing the video input to identify and locate a user's hand;determining that the user is performing the intended task; processingthe task data with the intended task to render a visualized job card onthe NTE display system; determining when there is a job card block,defined as when the visualized job card is in front of the hand; andremoving the visualized job card or a portion of the visualized job cardcausing the job card block when there is a job card block.

A system for augmented reality (AR) safe visualization for use with anear to eye (NTE) display system worn by a user is provided. The systemincludes: a controller comprising a processor programmed with an ARprogram and a task database storing task data; a camera mounted to theNTE display system and providing video input; a user input deviceproviding an intended task having an associated object; the processorreceiving the video input and coordinating the video image processing ofthe video input to identify therein a user's hand and the object; theprocessor receiving an intended task and retrieving task data includinga job card based thereon; the processor determining that the user isperforming the intended task; the processor processing the task datawith the intended task to render a visualized job card; the processordetermining when there is a job card block, defined as when thevisualized job card is in front of the hand, and removing the visualizedjob card or removing a portion of the visualized when there is a jobcard block.

Furthermore, other desirable features and characteristics of the systemand method will become apparent from the subsequent detailed descriptionand the appended claims, taken in conjunction with the accompanyingdrawings and the preceding background.

BRIEF DESCRIPTION OF THE DRAWINGS

The present application will hereinafter be described in conjunctionwith the following drawing figures, wherein like numerals denote likeelements, and:

FIG. 1 is a functional block diagram illustrating a system for augmentedreality safe visualization, in accordance with various exemplaryembodiments;

FIGS. 2-7 are illustrations showing several use cases for the system foraugmented reality safe visualization, in accordance with variousexemplary embodiments; and

FIG. 8 provides a flow chart for a method for augmented reality safevisualization, in accordance with various embodiments.

DETAILED DESCRIPTION

The following detailed description is merely illustrative in nature andis not intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Thus, any embodiment described herein as “exemplary” is not necessarilyto be construed as preferred or advantageous over other embodiments. Allof the embodiments described herein are exemplary embodiments providedto enable persons skilled in the art to make or use the invention andnot to limit the scope of the invention that is defined by the claims.Furthermore, there is no intention to be bound by any expressed orimplied theory presented in the preceding technical field, background,brief summary, or the following detailed description.

As mentioned, ensuring a user's safety during performance of a taskpresents a technological problem with the use of conventional ARsystems. As used herein, “AR visualization” and “visualization” areinterchangeable and visualization of items, such as, tools, objects, jobcards, hands, and the like, refers to using AR to render images of theitems on a see-through surface, such that the visualized items appear tothe user to be in the user's view of the outside world. Examples of thesee-through surface (FIG. 1, 20) include near to eye (NTE) displaysystems such as a lens of a pair of eyeglasses and a display windshield.Therefore, visualized items should not interfere with a user's vision orthe user's view of his actual hands, his actual tools, and the actualobject of an intended task. The provided systems and methods for AR safevisualization during performance of a task improve upon existing ARsystems by addressing these technological problems in an unconventionalway, in addition to providing other technological enhancements. Theprovided systems and methods for AR safe visualization duringperformance of a task may be used to provide visualization of differentparts of a maintained device, to highlight and distinguish interestingparts of it, to display a window or text box with a section ofmaintenance tasks (which may be referred to as a job card), to displaypages of a maintenance manual, and the like. The below descriptionprovides more detail as to these functions.

Turning now to FIG. 1, a functional block diagram of a system for ARsafe visualization during performance of a task 102 is depicted(shortened herein to AR system 102), in accordance with variousexemplary embodiments. The AR system 102 includes a source of videoinput 106, a user input device 108, modules for video image processing(including a hand detection module 110, an object detection module 112,and a tool detection module 114), a NTE display system 116, and an ARcontroller 104.

Depending on the embodiment, the user input device 108 may be realizedas a cursor control device (CCD), keypad, touchpad, keyboard, mouse,touch panel (or touchscreen), joystick, knob, line select key, voicecontroller, gesture controller, or another suitable device adapted toreceive input from a user. Non-limiting examples of user input receivedvia user input devices 108 include: selecting an intended task, loadingor updating AR program 162 and stored variables, providing confirmationsand/or cancellations for intended tasks, and loading and updating thecontents of the database 156, each of which is described in more detailbelow.

The AR controller 104 receives video input and coordinates video imageprocessing of the video input in order to identify and locate hands,objects, and tools. The algorithms used for video image processing maybe provided by commercially available video processing modules. In someembodiments, the video input processing may be performed by onemultifunctional algorithm in one multifunctional module. In otherembodiments, the video input processing may be performed by separatealgorithms. FIG. 1 depicts a separate hand detection algorithm in thehand detection module 110, an object detection algorithm in the objectdetection module 112, and a tool detection algorithm in the tooldetection module 114.

The NTE display system 116 may be deployed as a being part of any one ofa variety of wearable platforms, such as glasses, goggles, helmets, andother wearable display devices. The display system 116 may beimplemented using any one of numerous known see-through display surfaces20 suitable for rendering textual, graphic, and/or iconic information ina format viewable by a user. The display surface 20 may support ARvisualization using two-dimensional images, three-dimensional images,and/or animations. Accordingly, each display system 116 communicateswith the AR controller 104 using a communication protocol that is eithertwo-dimensional or three-dimensional, and may support the visualizationof text, alphanumeric information, objects, tools, visual symbology, andthe like, as described herein.

In various embodiments, the source of video input 106 is a cameramounted to the NTE display system or associated wearable platform that auser may wear. The source of video input 106 is understood to beoriented to capture video input inclusive of a field of view of the userwhen the user is wearing the wearable platform. The video input is avideo stream.

The AR controller 104 generates commands for the NTE display system 116to employ AR render visualized items on the see-through surface 20.Rendering a visualized item may include sizing and orienting thevisualized item to match the size, orientation, and perspective of anassociated actual item. For example, rendering a visualized bolt mayinclude sizing and orienting the visualized bolt so that it appears tofit right over the actual bolt; in a scenario like this, the visualizedbolt may be rendered in a visually distinguishable color, to draw theuser's attention to the location of actual bolt.

The AR controller 104 performs the functions of the AR system 102. Asused herein, the term “controller” may be interchanged with the term“module;” each refers to any means for facilitating communicationsand/or interaction between the elements of the AR system 102 andperforming additional processes, tasks and/or functions to supportoperation of the AR system 102, as described herein. In variousembodiments, the controller 104 may be any hardware, software, firmware,electronic control component, processing logic, and/or processor device,individually or in any combination. Depending on the embodiment, thecontroller 104 may be implemented or realized with a general purposeprocessor (shared, dedicated, or group) controller, microprocessor, ormicrocontroller, and memory that executes one or more software orfirmware programs; a content addressable memory; a digital signalprocessor; an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA); any suitable programmable logic device;combinational logic circuit including discrete gates or transistorlogic; discrete hardware components and memory devices; and/or anycombination thereof, designed to perform the functions described herein.

Accordingly, in FIG. 1, an embodiment of the AR controller 104 isdepicted as a computer system comprising a processor 150 and a storagemedia 152. The processor 150 may comprise any type of processor ormultiple processors, single integrated circuits such as amicroprocessor, or any suitable number of integrated circuit devicesand/or circuit boards working in cooperation to carry out the describedoperations, tasks, and functions by manipulating electrical signalsrepresenting data bits at memory locations in the system memory, as wellas other processing of signals. The storage media 152 may comprise RAMmemory, ROM memory, flash memory, registers, a hard disk, or anothersuitable non-transitory short or long-term storage media capable ofstoring computer-executable programming instructions or other data forexecution. In various embodiments, the storage media 152 may be locatedon and/or co-located on the same computer chip as the processor 150.Generally, the storage media 152 maintains data bits and may be utilizedby the processor 150 as storage and/or a scratch pad during operation.Information in the storage media 152 may be organized and/or importedfrom an external source during an initialization step of a process; itmay also be programmed via a user input device 108.

During operation of the AR controller 104, as well as the AR system 102,the processor 150 loads and executes one or more programs, algorithmsand rules embodied as instructions and applications, and specificallyloads and executes the AR program 162, contained within the storagemedia 152. The AR program 162 comprises rules and algorithms that, whenexecuted by the processor 150, determine the operations performed by theAR controller 104, such as “receive video input, receive intended task,reference task database, video process the video input, etc” (30). NovelAR program 162 and associated stored variables may be stored in afunctional form on computer readable media, for example, as depicted, instorage media 152. While the depicted exemplary embodiment of the ARcontroller 104 is described in the context of a fully functioningcomputer system, those skilled in the art will recognize that themechanisms of the present disclosure are capable of being distributed asa program product 166.

As a program product 166, one or more types of non-transitorycomputer-readable signal bearing media may be used to store anddistribute the program 162, such as a non-transitory computer readablemedium bearing the program 162 and containing therein additionalcomputer instructions for causing a computer processor (such as theprocessor 150) to load and execute the program 162. Such a programproduct 166 may take a variety of forms, and the present disclosureapplies equally regardless of the type of computer-readable signalbearing media used to carry out the distribution. Examples of signalbearing media include: recordable media such as floppy disks, harddrives, memory cards and optical disks, and transmission media such asdigital and analog communication links. It will be appreciated thatcloud-based storage and/or other techniques may also be utilized asstorage media 152 and as program product time-based viewing of clearancerequests in certain embodiments.

In various embodiments, the processor/memory unit of the controller 104may be communicatively coupled (via a bus 155) to an input/output (I/O)interface 154 and a task database 156. The bus 155 serves to transmitprograms, data, status and other information or signals between thevarious components of the controller 104. The bus 155 can be anysuitable physical or logical means of connecting computer systems andcomponents. This includes, but is not limited to, direct hard-wiredconnections, fiber optics, infrared and wireless bus technologies.

The I/O interface 154 enables intra-controller 104 communication, aswell as communications between the controller 104 and other AR system102 components, and between the controller 104 and any external datasources. The I/O interface 154 may include one or more networkinterfaces and can be implemented using any suitable method andapparatus. The I/O interface 154 can support a variety of types ofwireless communication, and can perform signal processing (e.g.,digitizing, data encoding, modulation, etc.) as is known in the art. Invarious embodiments, the I/O interface 154 is configured to supportcommunication from an external system driver and/or another computersystem. Also, in various embodiments, the I/O interface 154 may supportcommunication with technicians, and/or one or more storage interfacesfor direct connection to storage apparatuses, such as the task database156.

The task database 156 is a storage media that has stored therein aplurality of tasks. The tasks may be stored as a data string, having atag portion that is the task name, and then having fields for anassociated: object; job card; tool; etc. If a task does not have anassociated tool, that field may be left empty. If, for a given fieldname, a task has more than one associated entry, the additional entriesmay be stored in sequence (for example, a first job card, and a secondjob card). In an embodiment, each task of the plurality of tasks has atleast one associated job card. In an embodiment, each task of theplurality of tasks has an associated object and at least one associatedjob card. As mentioned, a job card is a piece of instruction, and itsformat may be alphanumeric, and/or include a symbolic image, and/orinclude an animation. In various embodiments, some tasks of theplurality of tasks also have an associated tool. For example, for atask, “refill the oil,” the associated object may be a vehicle engine,and the associated first job card may instruct, “unscrew the oil cap.”In this example tasks, there may not be an associated tool. In anotherexample, the task is “change the oil,” the object is again the vehicleengine, and the task may have an associated tool, such as a wrench.Additionally, the task “change the oil” may have multiple sequential jobcards, such as “unscrew an oil drain plug,” “wait until oil has stoppedflowing out,” etc. In various embodiments that have multiple job cards,the format of each job card may or may not be different than the formatof those preceding it or succeeding it. The task database 156 is shownas being internal to the AR controller 104, however in otherembodiments, task database 156 may be external to the AR controller 104.

It will be appreciated that the AR system 102 may differ from theembodiment depicted in FIG. 1. Having described the functional blockswithin the AR system 102, we move to FIGS. 2-7 for illustrations ofvarious use cases and to FIG. 8 for an exemplary embodiment of a method800 associated with the AR system 102.

For illustrative purposes, the following description of method 800 mayrefer to elements mentioned above in connection with FIG. 1. Inpractice, portions of method 800 may be performed by differentcomponents of the described system. It should be appreciated that method800 may include any number of additional or alternative tasks, the tasksshown in FIG. 8 need not be performed in the illustrated order, andmethod 800 may be incorporated into a more comprehensive procedure ormethod having additional functionality not described in detail herein.Moreover, one or more of the tasks shown in FIG. 8 could be omitted froman embodiment of the method 800 as long as the intended overallfunctionality remains intact.

The method starts, and at 802 the AR controller 104 is initialized. Asmentioned above, initialization may comprise uploading or updatinginstructions and applications, the program 162, and the task lookuptables stored in the database 156.

At 804 video input is received, and at 806 the intended task isreceived. As mentioned, the intended task may be provided as user inputfrom the user input device 108. In an embodiment, the intended taskincludes an actual object. At 808, the method 800 uses the intended taskinput to retrieve from the task database 156 task data, which includesat least a job card. In some embodiments, the task data includes a jobcard and a tool. In some embodiments, the task data includes a job card,a tool, and an object. In various embodiments, the intended task, whenmatched to tasks in the task database, retrieves multiple job cards,and/or multiple objects, and/or multiple tools.

At 810, the method 800 performs video image processing on the videoinput to identify and locate the user's hand (ie., the actual hand). Insome embodiments, at 810, the method 800 performs image processing onthe video input to identify and locate the user's hand and the actualobject. In some embodiments, at 810, the method 800 performs imageprocessing on the video input to identify and locate the user's hand,the actual object, and an actual tool. At 812, the method 800 processesthe intended task with the retrieved task data and renders on thesee-through screen 20 of the display system 116, one or more of: avisualized job card, a visualized tool, and a visualized object. In anembodiment, at 812, a visualized job card and a visualized object arerendered. In an embodiment at 812, a visualized job card, a visualizedtool, and a visualized object are rendered. At this point, the user'sfield of view through the surface 20 includes the outside world plus theone or more visualized items.

At 814, the method 800 determines that the user is performing theintended task. In some embodiments, the AR controller 104 determineswhen the user is performing the task by detecting that the user hasapproached the object of the task (in this case, the engine 204). Insome embodiments, the AR controller 104 determines when the user isperforming the task by receiving user input so stating. The depictedfield of view areas of FIGS. 2-7 are representative of potential viewsat operation 814.

As shown in FIG. 2, area 200 represents a user's field of view. Theuser's actual left hand 202 is near the engine 204, and a job card 206is visualized. In an embodiment, the visualized the job card 206 appearslike a text block on the left side of the user's view of the outsideworld and engine 204. In an embodiment, the job card 206 takes up lessthan one sixth of the overall area 200. In various embodiments, the jobcard 206 includes textual instructions 208, and a page indicator 210(indicating that this is page 3 of 6 total job cards). In variousembodiments, the job card 206 includes simple pictorial instructions. Ascan be observed in FIG. 2, the job card 206 is behind the user's hand202 and occludes some of the user's view of the left portion of theengine 204.

At some point during the performance of the task, the user's hand 202may appear to go behind the visualized job card 206. This is depicted inFIG. 3, field of view area 300. The user's view of his hand is now atleast partially blocked by the visualized job card 206, this is a jobcard block. At operation 816, the method 800 determines whether theuser's hand or the user's actual tool is behind the visualized job card206, (job card block) and if it is, at 818, the visualized job card 206may be removed from the image (FIG. 4), it may be moved out of theuser's view of his hand 202 and/or tool and the object of his task (FIG.5), or a portion of the job card 206 responsible for the job card blockmay be removed. In FIG. 4, responsive to determining that there was ajob card block, the AR controller 104 has removed the visualized jobcard 206, as shown in field of view area 400. As mentioned, in variousembodiments, responsive to determining that there was a job card block,the AR controller 104 instead removes a portion of the visualized jobcard 206, that is in front of the user's hand or actual tool. In FIG. 5,field of view area 500, responsive to determining that there was a jobcard block, the AR controller 104 has moved the visualized job card 206up and away from the user's hand 202, and away from the engine 204. Saiddifferently, the visualized job card 206 is moved to a place on thesee-through screen 20 on display system 116 in which it does not blockthe user's view of his hand 202 or of the engine 204. In variousembodiments, moving the visualized job card 206 further includesrotating it clockwise or counter-clockwise, or changing its perspectiveto appear to extend lengthwise from closer to the user's eyes to fartherfrom the user's eyes.

In summary, when there is a job card block (operation 816, FIG. 8), thecontroller 104 may respond by: removing the visualized job card,removing a portion of the visualized job card in front of the hand, ormoving the visualized job card.

In the example shown in FIGS. 6-7, the rendering of visualized objectsand visualized tools is described. In the examples of FIGS. 6-7, theintended task is loosen a bolt 608, which includes the task and theobject (the bolt), and the retrieved task data further includes using awrench to loosen the bolt 608. As the user approaches the engine 204,the AR controller may render visualized wrench 606. As can be seen, thefield of view area 600 shows the visualized wrench 606 fitted exactly tothe bolt 608 and engaged with it. The visualized wrench 606 is not thesame as an actual wrench, but it is rendered to be proportionate and ina perspective view. In various embodiments, the visualized wrench 606 isrendered with animation to show the turning movement that will berequired to accomplish the task (e.g., counter-clockwise movement). Inpractice, responsive to this view 600, the user would likely obtain theactual wrench 702 and begin to move it to match a location and movementindicated with the visualized wrench 606. In various embodiments, ifFIG. 6 were rendered in color, the visualized bolt that overlays actualbolt 608 might be treated as a visualized object, and such that it isrendered in a visually distinctive color with respect to the remainderof the engine 204 (for example, the visualized bolt may be red).

Moving now to FIG. 7, as mentioned herein, when the user begins theintended task, it is desirable for the user's view of his hands, hisactual tool (the actual wrench 702), and the actual object that he isworking on (i.e., the bolt 608 and/or the engine 204) to be unobscured.Accordingly, at 820, the method 800 determines whether the user's handis behind the visualized tool 606 (a tool block) and if it is, at 822,one of the following is done by the controller 104: the visualized tool606 is removed from the image (FIG. 7), a portion of the visualized tool606 that blocks the user's hand or object of user's work is removed, orthe visualized tool 606 is moved out of the user's view of his hand andthe object he is working on. In various embodiments, at 824, the method800 determines whether the user's hand or a user's actual tool is behinda visualized object (for example, the bolt 608, an object block) and ifit is, at 824, the visualized object 608 is either removed from theimage, a portion of the visualized object 608 responsible for theblocking is removed, or the visualized object 608 is moved out of theuser's view of his hand and the object of his task.

In summary, when there is a tool block (operation 820, FIG. 8), thecontroller 104 may respond by: removing the visualized tool, removing aportion of the visualized tool in front of the hand, or moving thevisualized tool. Likewise, when the controller 104 determines that thereis an object block (operation 824, FIG. 8), the controller 104 mayrespond by: removing the visualized object, removing a portion of thevisualized object in front of the hand, or moving the visualized object.In this manner, a user can be assisted in performing a technical taskvia the information associated with the visualization of certain itemswithout the performance of this technical task being obscured, by thesevisualized items, from the user's view.

When the user's hand 202 is not blocked by visualized items (visualizedtools, job cards, or objects, meaning there is no job card block, notool block, and no object block) the method returns to 810 to continuevideo processing the video input. Also, when the user's actual tool isnot blocked by visualized tools, job cards, or objects, the methodreturns to 810 to continue video processing the video input. Afterremoving or moving visualized images of job cards (at 818), objects,and/or tools (at 822), the method may end or return to 810. When themethod returns to 810 after having previously removed a visualized item,the AR controller 104 may determine that the visualized item or portionof visualized item would no longer cause a block and re-render themissing portion of the visualized item or the entire visualized item.

In various embodiments, at operation 816, the method 800 additionallyidentifies a job card block as including when the user's actual tool(e.g., wrench 702) is behind the visualized job card 206, and if it is,at 818, the visualized job card 206 is either removed from the image, orit is moved out of the user's view of his actual tool and the object heis working on. Said differently, the visualized job card 206 is moved toa place on the see-through screen 20 on display system 116 in which itdoes not block the user's view of his actual tool (e.g., wrench 207) orof the engine 204.

In various embodiments, at operation 820, the method 800 additionallyidentifies a tool block as including when the user's actual tool (e.g.,wrench 702) is behind the visualized tool 606, (tool block) and if itis, at 818, the visualized tool 606 is either removed from the image, orit is moved out of the user's view of his actual tool and the object heis working on.

As is readily appreciated, the above examples are non-limiting, and manyothers may be addressed by the AR controller 104. Thus, systems andmethods for enhanced computer vision systems have been provided.

Those of skill in the art will appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Some ofthe embodiments and implementations are described above in terms offunctional and/or logical block components (or modules) and variousprocessing steps. However, it should be appreciated that such blockcomponents (or modules) may be realized by any number of hardware,software, and/or firmware components configured to perform the specifiedfunctions. To clearly illustrate the interchangeability of hardware andsoftware, various illustrative components, blocks, modules, circuits,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present invention. For example, anembodiment of a system or a component may employ various integratedcircuit components, e.g., memory elements, digital signal processingelements, logic elements, look-up tables, or the like, which may carryout a variety of functions under the control of one or moremicroprocessors or other control devices. In addition, those skilled inthe art will appreciate that embodiments described herein are merelyexemplary implementations.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general-purpose processor may be a microprocessor, but in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The steps of a method or algorithm described in connection with theembodiments disclosed herein may be embodied directly in hardware, in asoftware module executed by a controller or processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumknown in the art. An exemplary storage medium is coupled to theprocessor such that the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. The processor and the storagemedium may reside in an ASIC.

In this document, relational terms such as first and second, and thelike may be used solely to distinguish one entity or action from anotherentity or action without necessarily requiring or implying any actualsuch relationship or order between such entities or actions. Numericalordinals such as “first,” “second,” “third,” etc. simply denotedifferent singles of a plurality and do not imply any order or sequenceunless specifically defined by the claim language. The sequence of thetext in any of the claims does not imply that process steps must beperformed in a temporal or logical order according to such sequenceunless it is specifically defined by the language of the claim. Theprocess steps may be interchanged in any order without departing fromthe scope of the invention as long as such an interchange does notcontradict the claim language and is not logically nonsensical.

Furthermore, depending on the context, words such as “connect” or“coupled to” used in describing a relationship between differentelements do not imply that a direct physical connection must be madebetween these elements. For example, two elements may be connected toeach other physically, electronically, logically, or in any othermanner, through one or more additional elements.

While at least one exemplary embodiment has been presented in theforegoing detailed description of the invention, it should beappreciated that a vast number of variations exist. It should also beappreciated that the exemplary embodiment or exemplary embodiments areonly examples, and are not intended to limit the scope, applicability,or configuration of the invention in any way. Rather, the foregoingdetailed description will provide those skilled in the art with aconvenient road map for implementing an exemplary embodiment of theinvention. It being understood that various changes may be made in thefunction and arrangement of elements described in an exemplaryembodiment without departing from the scope of the invention as setforth in the appended claims. It will also be appreciated that while thedepicted exemplary embodiment is described in the context of a fullyfunctioning computer system, those skilled in the art will recognizethat the mechanisms of the present disclosure are capable of beingdistributed as a program product with one or more types ofnon-transitory computer-readable signal bearing media used to store theprogram and the instructions thereof and carry out the distributionthereof, such as a non-transitory computer readable medium bearing theprogram 162 and containing computer instructions stored therein forcausing a computer processor (such as the processor 150) to perform andexecute the program 162. Such a program product may take a variety offorms, and the present disclosure applies equally regardless of theparticular type of computer-readable signal bearing media used to carryout the distribution. Examples of signal bearing media include:recordable media such as floppy disks, hard drives, memory cards andoptical disks, and transmission media such as digital and analogcommunication links. It will be appreciated that cloud-based storageand/or other techniques may also be utilized in certain embodiments.

1. An augmented reality (AR) safe visualization method for use with anear to eye (NTE) display system worn by a user, comprising: at aprocessor associated with the NTE display system and programmed with anAR program, receiving video input; receiving an intended task;retrieving, for the intended task, an object, a tool, and a job card,from a task database; rendering a visualized job card, visualizedobject, and a visualized tool on the NTE display system; video imageprocessing the video input to identify and locate a user's hand and theobject; determining that the user is performing the intended task;determining when there is a job card block, defined as when thevisualized job card is in front of the hand; determining when there is atool block, defined as when the visualized tool is in front of the hand;determining when there is an object block, defined as when thevisualized object is in front of the hand; removing the visualized jobcard, or a portion of the visualized job card in front of the hand, whenthere is a job card block; removing the visualized tool, or a portion ofthe visualized tool in front of the hand, when there is a tool block;and removing the visualized object, or a portion of the visualizedobject in front of the hand, when there is an object hand block.
 2. Themethod of claim 1, further comprising: video image processing the videoinput to identify and locate a user's actual tool; and identifying a jobcard block as including when the user's actual tool is behind thevisualized job card.
 3. The method of claim 2, further comprisingidentifying a tool block as including when the user's actual tool isbehind the visualized tool.
 4. The method of claim 3, furthercomprising: determining that a previously removed visualized item orportion of visualized item would no longer cause a block; andre-rendering the visualized item or portion of visualized item.
 5. Amethod for augmented reality (AR) safe visualization for use with a nearto eye (NTE) display system worn by a user, comprising: at a processorassociated with the NTE display system and programmed with an ARprogram, performing the operations of: receiving video input from acamera mounted to the NTE display system; receiving an intended taskfrom a user input device; retrieving task data from a task databasebased on the intended task; video image processing the video input toidentify and locate a user's hand; determining that the user isperforming the intended task; processing the task data with the intendedtask to render a visualized job card on the NTE display system;determining when there is a job card block, defined as when thevisualized job card is in front of the hand; and removing the visualizedjob card or a portion of the visualized job card causing the job cardblock when there is a job card block.
 6. The method of claim 5, furthercomprising: video image processing the video input to identify andlocate a user's actual tool; and identifying a job card block asincluding when the user's actual tool is behind the visualized job card.7. The method of claim 6, further comprising: processing the task datawith the intended task to render a visualized tool; determining whenthere is a tool block, defined as when the visualized tool is in frontof the hand; and removing the visualized tool or a portion of thevisualized tool causing the tool block when there is a tool block. 8.The method of claim 7, further comprising identifying a tool block asincluding when the user's actual tool is behind the visualized tool. 9.The method of claim 8, further comprising: processing the task data withthe intended task to render a visualized object; determining when thereis an object block, defined as when the visualized object is in front ofthe hand; and removing the visualized object when there is an objecthand block.
 10. The method of claim 9, further comprising identifying anobject block as including when the user's actual tool is behind thevisualized object.
 11. The method of claim 10, further comprising:determining that a previously removed visualized tool or portion of avisualized tool would no longer cause a tool block; and re-rendering thevisualized tool.
 12. The method of claim 10, further comprising:determining that a previously removed visualized job card or portion ofa job card would no longer cause a job card block; and re-rendering thevisualized job card.
 13. A system for augmented reality (AR) safevisualization for use with a near to eye (NTE) display system worn by auser, comprising: a controller comprising a processor programmed with anAR program and a task database storing task data; a camera mounted tothe NTE display system and providing video input; a user input deviceproviding an intended task having an associated object; the processorreceiving the video input and coordinating the video image processing ofthe video input to identify therein a user's hand and the object; theprocessor receiving an intended task and retrieving task data includinga job card based thereon; the processor determining that the user isperforming the intended task; the processor processing the task datawith the intended task to render a visualized job card; the processordetermining when there is a job card block, defined as when thevisualized job card is in front of the hand, and removing the visualizedjob card or removing a portion of the visualized when there is a jobcard block.
 14. The system of claim 13, wherein the processor is furtherprogrammed to: video image process the video input to identify andlocate a user's actual tool; and identify a job card block as includingwhen the user's actual tool is behind the visualized job card.
 15. Thesystem of claim 14, wherein the processor is further programmed to:process the task data with the intended task to render a visualizedtool; determine when there is a tool block, defined as when thevisualized tool is in front of the hand; and remove the visualized toolor remove a portion of the visualized tool causing the tool block, whenthere is a tool block.
 16. The system of claim 15, wherein the processoris further programmed to identify a tool block as including when theuser's actual tool is behind the visualized tool.
 17. The system ofclaim 16, wherein the processor is further programmed to: process thetask data with the intended task to render a visualized object;determine when there is an object block, defined as when the visualizedobject is in front of the hand; and remove the visualized object orremove a portion of the visualized object causing the object block, whenthere is an object block.
 18. The system of claim 17, wherein theprocessor is further programmed to identify an object block as includingwhen the user's actual tool is behind the visualized object.
 19. Thesystem of claim 18, wherein the processor is further programmed to:determine that a previously removed visualized tool would no longercause a tool block; and re-render the visualized tool or portion of thevisualized tool that was removed.
 20. The system of claim 19, whereinthe processor is further programmed to: determine that a previouslyremoved visualized job card would no longer cause a job card block; andre-render the visualized job card or portion of the visualized job cardthat was removed.